<html>
<head><meta charset="utf-8"><title>parallel lintcheck · clippy · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/index.html">clippy</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html">parallel lintcheck</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="226883905"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226883905" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226883905">(Feb 18 2021 at 21:28)</a>:</h4>
<p><span class="user-mention" data-user-id="264664">@flip1995</span> in other news, I managed to parallelize lintcheck so that we can <code>cargo clippy</code> several crates at a time!<br>
The biggest problem was that we can't continue sharing our cargo-clippy-on-source target dir because cargo would lock and kill the parallelism <span aria-label="big smile" class="emoji emoji-1f604" role="img" title="big smile">:big_smile:</span> <br>
I've resorted to using <code>${N_CPUS}</code> target dirs at the same time (one for each cpu-core ideally)  which is not optimal for caching but still better than nothing.</p>



<a name="226884408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226884408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226884408">(Feb 18 2021 at 21:32)</a>:</h4>
<p>I see. Sacrificing memory for speed. The modern software engineering way. <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span> But that is indeed great news!</p>



<a name="226884489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226884489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226884489">(Feb 18 2021 at 21:33)</a>:</h4>
<p>Just finished running on the ~400 crates-io crates.toml that I added to your repo.<br>
The resulting log file is 35 megabyte lol</p>



<a name="226884717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226884717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226884717">(Feb 18 2021 at 21:35)</a>:</h4>
<blockquote>
<p>I see. Sacrificing memory for speed. </p>
</blockquote>
<p>also disk space...<br>
13 gigs in the target dir...<br>
We might need to auto-clean that every then and now</p>



<a name="226885282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226885282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226885282">(Feb 18 2021 at 21:39)</a>:</h4>
<p>Oh that is really big :O Yeah, I don't think we can use this for triaging new lints. But it is definitely great to find ICEs and prevent really bad bugs in Clippy.</p>



<a name="226885535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226885535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226885535">(Feb 18 2021 at 21:41)</a>:</h4>
<p>oh I have a bug where it creates one target dir per crate <span aria-label="rolling eyes" class="emoji emoji-1f644" role="img" title="rolling eyes">:rolling_eyes:</span>  ...</p>



<a name="226895986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226895986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226895986">(Feb 18 2021 at 22:53)</a>:</h4>
<p>So it seems that for the current set of crates, using two target dirs and running two clippys at the same time is all in all slower than using a single target dir and only sequential clippy :(</p>



<a name="226896075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226896075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226896075">(Feb 18 2021 at 22:54)</a>:</h4>
<p>I did not think that the shared target dir would make such a difference</p>



<a name="226896362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226896362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226896362">(Feb 18 2021 at 22:56)</a>:</h4>
<p>I think the big difference is, that cargo already uses all CPUs to compile deps in parallel. And then with a shared target dir it just reuses the deps from the previous crate.</p>



<a name="226896600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226896600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226896600">(Feb 18 2021 at 22:58)</a>:</h4>
<p>yes</p>



<a name="226896655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226896655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226896655">(Feb 18 2021 at 22:59)</a>:</h4>
<p>so it kind of depends on how many deps the average crate that we check has.<br>
The less deps it has, the more of an impact the parallelization will have</p>



<a name="226896790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226896790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226896790">(Feb 18 2021 at 23:00)</a>:</h4>
<p>But I also only have a 2c/4t system, I wonder how it would look on let's say a quad or octacore system</p>



<a name="226897025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226897025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226897025">(Feb 18 2021 at 23:02)</a>:</h4>
<p>I could try that tomorrow. I have access to a 12t machine (and some 40c+ servers at work, I may be able to abuse after working hours <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span>).</p>



<a name="226897285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226897285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226897285">(Feb 18 2021 at 23:04)</a>:</h4>
<p>:o</p>



<a name="226897729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226897729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226897729">(Feb 18 2021 at 23:09)</a>:</h4>
<p>but then again, the more crates we check in total, the more likely we are going to have a random "cache hit" ....</p>



<a name="226898622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226898622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226898622">(Feb 18 2021 at 23:17)</a>:</h4>
<p>I can time it with different number of crates, if you send me your patch for parallelization.</p>



<a name="226898816"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226898816" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226898816">(Feb 18 2021 at 23:20)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust-clippy/pull/6764">https://github.com/rust-lang/rust-clippy/pull/6764</a></p>



<a name="226904412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226904412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226904412">(Feb 19 2021 at 00:23)</a>:</h4>
<p>ran on the 400-crates toml now:<br>
master: <code>1458,48s user 109,25s system 76% cpu 33:58,81 total</code><br>
rayon par_iter: <code>1873,03s user 137,84s system 144% cpu 23:13,41 total</code></p>



<a name="226912147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226912147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226912147">(Feb 19 2021 at 02:15)</a>:</h4>
<p>personally as a clippy user, I would really hate to rebuild the shared cache over and over again</p>



<a name="226912171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226912171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226912171">(Feb 19 2021 at 02:15)</a>:</h4>
<p>In particular I would run out of memory really quickly and start thrashing</p>



<a name="226912264"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226912264" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226912264">(Feb 19 2021 at 02:17)</a>:</h4>
<p>I encourage you to use <a href="http://docs.rs">docs.rs</a> as a test case, I think you'll find several caches takes <em>much</em> more time than running clippy in sequence regardless of the number of cores</p>



<a name="226923540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226923540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226923540">(Feb 19 2021 at 05:57)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> this is only a dev tool <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="226923548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226923548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226923548">(Feb 19 2021 at 05:57)</a>:</h4>
<p>Ah ok, sorry for the noise</p>



<a name="226923621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226923621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226923621">(Feb 19 2021 at 05:58)</a>:</h4>
<p>it does not change how  <code>cargo clippy</code> works :)</p>



<a name="226994005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/226994005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#226994005">(Feb 19 2021 at 17:00)</a>:</h4>
<p>See my timing results in the PR. TL;DR: parallelizing  this makes sense if you have access to a machine with <strong>many</strong> cores, so I would just add a flag that makes this possible.</p>



<a name="227039973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/parallel%20lintcheck/near/227039973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matthiaskrgr <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/parallel.20lintcheck.html#227039973">(Feb 19 2021 at 22:30)</a>:</h4>
<p>added the flag now (pr also includes a few other minor changes)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>